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■PREFACE ' ^ 

Issues and Recommendations i$ the Introductory section of a , 
seven-part report on the evaluation of civilian training in 
the Federal government. The remaining six parts are case 
fstudies which derapnst rate the conduct .of evaluation across 
la broad range of tipaining: E^ch case study has been. ■ 
written as an example of how eyaluat9rs inay^egotiate, 
design, develpp, arfd implement ,t raining evaluation in an » 
agency setting. .Each is also am example of how an evaluation 
effort may be reported jt<) a decision-maker.. ' 

... ■ ^ \ j[ , . ■ I y ' ' . -.^ \ " ■ 

'Much ejpfort was expended iYi each demonstration project to use as ^ 
comprehensive an evaluation design* as possible,' given the organ- 
izational constraints^ whic^i confronted eacji evaluat.ion team. ' 
Emphasis, was ^given* tO' the utility pf the^ results. We asked in. H * 
each case what it was- that the. client most ne^ed to know about . , 
the training being evaluated. We discovered J^sthand wh^t pro- 
i^rfirn evaluators have dj-scussed in the professle^nal literature on - 
et?aluation research: comprehensive e.valuatlon,i|esign can^ be . 
difficult at best to achieve in a normal work getting, and utility 
can be difficult .to define in advance. Yet, we believe ovir efforts 
have yielded information that, is basically valid and usieful to 
our clients. • ' . • . ^ 

We wish publicly to express thanks for the' cooperation and 
assistance ptovidecl in each of the iix agenci^es whojse course or 
program was evaluated. Without people who were will injg not only 
to have a portion df their training thoroughly assessed but to 
collaborate actively in that ef!fort, *theffe'^emQnstrations would 
not have been, feasible. ^ 

■ ■' ' ■ . ■ ' ' . .- f.'"^'' ■ 

We anticipate that the cooperation agencies demon^rated ^uring 
this project along with the results of the evaluations* wljd prpmpt 
further efforts to systematically evaluate training in the'Federal 
government. Perhaps more significant t;han any* of the reported 
ftmllngs on training is the confirmation throiifeh this experience 
that comprehensive evaluation can be conducted, and that it can 
-lead to constructive recorariiehdations for change ^without intimations 
of failure. . ' r . ' *•■ 

The seven parts which cgijppri'se the report are:- \, v ' ^ 

Issyes and Recofi[^ndations . . ' / 
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INTRODUCTION - ■ * , ' \, 

• ■ # .. ' . ^. . . -.. •• •■' * ■ ■ , ' ' 

'in July 1978, following extended discussion with 0MB stdff, 0PM 
(then CSC) undertdok a training evaluation- demonstration project 
which involved the cooperation of six govemmental agencies. The 

• specif ications for , the project >ere forinalized by letter of . ' 
detail; .The essentials of that agreement were that CSC would >• 
undertake as felpresentative a sampling as possible -of Federal 

..dtrainitlg and would report 'it,iB conclusions and recommendations ■ 
regarding the evalxiation of Federal' training to 0MB by March 31, 

-19J9. .. ■ . • . '.. • • : " ■ \x ; ' j 

The demon strait ion evaluations have been, documented in thye^case , 
studies of this'.report • It; is our. intention to publish these 
training evaluatiqhs for dissemination po Federal agencies andv 
other organizations concerned' with accountability in PSderal ^ v . 
training. Ve bel^ieve-ithese evaluatl^ons can assist trainers ^rid; 
managers to develop and conduct their own evaluations of training;/v 
an<frbthex employee development * activi,ti^. *^.> 

: ■ . «^ ■ > ,'■.' ■ 

The Scope and Nature pf Federal^ Training ' 

In fiscal year 1977 , 555,544 civilian employees recein^ed 37>469,999 
hours of training at a reported cost of $256 , 941 ,.055 'Xexc luding 
trainees' salaries and certain other costs). Although this training 
represents an annual multi-million dollar e'xpenditure, the great 
bulk of it involves many relatively small expenditures and small 
numbers plE employees. Moreover, the manner of funding training^, 
the structure of training organizations ahd the resources available 
to those organizations vary widely. ' ^ 

The Evaluation Demonstrations - ~ - ^ f 

In the interest of obtaining as' representative a sample of Federal 
training as possible for the demonstration study, OPM*splected 
candidates whose course or program exemplified the following types 
of tf*aining: executive, managerial, technical, clerical and 
scientific (See Appendix for executive summaries). The j following 
Is a descriptive list of the proj'ects: ■ 

Agency ' Type of Training Specific Target 



Science and / Executive The Executiye 

Education ' Development 
Administration, * Program- 

USDA 



ERIC 



Fort Belvoir, 
Dept. of the Array 



Governraeiit Printing 
Office ' ^ 



Office of Persbnhbl 
Management 



National Aeronautics 
and Space Adrainisdra- 
tibn 



National Institutes 
of Health , 



Managera^nt 



Contracted TecHnlcal 
Interagency TechnicaT 



Clerical 



Technical / Sclent i f ic 



, A*raanagerl,al 
communications 
course for 
directorate 

A contracted- ^. 
photocomposition 
"course^ for 
6p€uratprQ 

An interagency 
course on ^ 
evaluation of 
-training courses 

' A workshop on 
travel regulations 
for secretaries .who 
prepare travel . 
orders and vouchees 

A course on radio- 
logical health 'for 
V radionuclide users 



The project served two basic purpQses. One was to produce examples 
of training; evaluation which would be useful to agency trainers 
and evaluators. ^Another /was to. generate recommendations • for 
improving. The practice of training evaluation as a means of improving 

Federal training. . ' * :^ • . 

\ ■ ■ ■ . . > ' ■ |1 • ■ ' • 

• ■ , ■» . . y . ■ ' • • », 

Train ing Evaluation; An Operational Definition 

By the term training cSyaluatfon we mean a systematic investigation 
of the value of a course or a program of l:raining or employe^ 
tievelopraent. Wetview the purpose of ' training evaluation a 
essentially two-fold: to yield judgment regarding how wel 
coiir«e or program met Its instructional objectives; and to 
Ji/flgment regarding the impact of the course or program on 
system, e.g., an entire organization, J 

We use the term sys tematic to emphasize an eSseijJ^i?^ difference 
between evaluation and assessment. ^We understand assessment as 
meaning the process' of measuring the effect of one variable or 
set of variables on another. Thus one^may Assess the impact of 
' training' on job perf ormancie. Evaluation, by way 6f contrast^ 
would . involve drawing inferences from the data which such an 
assessment migHt yield and weighing It. with other data sufch as^ 



er 



that generated 'by investigating the impact of other variables 
on job performance (to Include incentives, disincentives, work- 
flow and communications). . ^ 

The evaluation^demonstraCldns completed by 0PM 'then are 'systematic 
investigations of the value of an executive development; program 
and several training courses • As such these investigations impel 
the evaluators to, do more than merely collect and display data; \ 
tjliey x^uire the eVjgiluators to draw inferences a*out relative 
value in terms of co&os and benefits to the organization. 

It is .anticipated that the six evaluation demonstrations will/Tprove 
use£ul to .agency trainers as examples of. how evaluations can be 
negotiated, planned, conducted and reported in opei:ational -envi- 
ronments. However, the demonstrations are not meant to be applied ^ 
as strict models or paradigms. Each agency's training, courses 
and programs require' evaluation designs and methodology *whtch are 
tailored to meet the unique objectives, constraints and resources^ 
within that agency 

Pr oject Costs ' ' ' • . 

The salary cost incurred 'by' OPM^s Training Leadership Division in 
undertaking the project was approximately -$13Z, 000. This coat was 

^derived from a careful recording of hours spent by .each Itidlfvtdual 
on the project. Other cost not reco-rded but incurred' are those 

.associated with duplication of typed' ma ferial, use of computer time 
travel, st^rt-up time j^ot off ically charged to the project,, 
library resource material and ti,me given to the evaluation effort 
by ajgency personnel cooperating with the 0PM evaluation teams* 

Each evaluation 'demonstration was co'sted separately . (a detailed 
account of costs is given in ah appendix to each volume). The 
reader should consider these* costs ds suggestive rather than 
prescriptive. Irf some cases', instruments tehtral X^o the data 
collection prbcless had already been deyeXoped and validated 
before the evaluation, began. Had theism instruments been 
developed as part of thie evaluation, the^,^roject cost would . , 
have been.- increased Vignif leant ly. . - 

In addition to considering these co^ts suggestive, the reader . 
should. also^ consider that each evaluation effort can be seen as 
the first phase of an on-going evaluation process* , Should any 
of the agencies whose training we evaluated decide to continue 
the evaluation- process , their costs would be less than ours 
since some functions of evaluation may not have to be iterated, 
such as negotiating, establishing objectives , developing an 
evaluation design or developing basic instrumentation. 



While this paper is primarily cqncerned- *ith unresolved issues • 
related to the evaluation of training, th^re are certain fundamental 
conclusions that can be drawn from the t)roject. They are of 
sufficient importance that they are presented here: ^ 

1> Constructive evaluation o"f all otypes of training 
affordeti Federal civilian employees is well 
' ^^tth'in the present state of tl/e art;. 



2. 



Evalu3tion, applied to carefully selected targets^ 

reduce costs and improve the results of 
training significantly. 




3. Linking training with other problem-solving 

.approaches cair ptoduce benefits exceeding those , 
to be gained by methods solely concerned with 
improving the effectiveness of training. 

in all six demonstratigns we were able to draw reasonable, if 
sometimes guarded, conclusions about the effectiveness of training. 
We also discovered many impediments to drawing more unequivocal, 
far-reaching conclusions. However, i"-P°8t/ases, we were^able 
to project ways that those impediments could be overcbme, whether 
they involved selection of trainees, evaluation design or 
instrumentation for data-gathering. We therefore conclude that 
constructive evaluation, that which yields useful information 
to trainers and decisiU-makers, is clearly achievable. 

We were kble al^o to project ^peeific ways in which proposed 
changes would either directly reduce costs of training, improve 
the quality of the training, or achieve a greater impact on the 
mission of Che organization. Th*us we conclude that evaluation 
can lead to Improved economic efficiency ,by helping managers 
to reach their specific work objective? at relatively lower 
costs. " ^ • . 

Perhaps most linportantly , we found repeated evidence that 
training alone would npt solve many performance problems; This 
finding strengthens our conviction that training^ust be 
conceptualized as 'one' of a number of management tools for 
increasing the Ifficiejit use of labor- as a factor of production. 
A decision- to train should be r^eached after ■ consideration of . 
its- potential for effecting change relative to the cost and 
potential of other tools (communications, job redesign, job 
"aides, incerttive sytems', and perfoiUliance feedback, to mention 
a few). ' * 



ISSUES AND. RECOMMENDATIONS^' 



The project served to sliarpen the focus! on several Issues, which 
require the attention of 0MB and 0PM management. These issues- 
are addressed below. Each is briefly discussed in terms of its 
general nature an4 in most cases is exemplified by a -case in 
point« Potential solutiori^ of the issue are ..described along' with 
necessary conditions for those solutions to be. effective. 

Finally, recomi^endations for OMBf^nd *OPM actions are given. 

Issue 1: A pervasive lack of planning and analysis before 



sta^rting to train characterizes the approach to 
training ifl^many organizations. 



Discussion: 

■ • . • ' ' ■ 

a. ^ Training is often undertaken as the sole solution 'to a - 
perceived, prbblem when lack of skill or knowledge on the part 
of employees is only a portion of the problem. 

In these and other evaluations conducted by the training 
leadership staff, we have discovered situations where traiiling 
was initiate* as the only isolution to a problem when, if 
•training were 100% effective, only ^ fraction of the problem 
would be solved. . An /example of this was found in the travel 
regulations training we evaluated for this report. We 
discovered that although the training was planned and con- 
ducted in a highly competent mannexi, it could not solve 
performance problems which revolved around lack of incentives 
for satisfactory performance and the infrequency with Vhich 
many, secretaries prepare voucher^. We concluded that job/ 
aides would ameliorate performance as long as regular clerical 
personnel were [^eparing vouchers. "^We also concluded that 
shifting voucher preparation to the central finance office's 
travel section would be a potentially cost-effective solution. 

In contrast, we foiind during our evaluation of a. managerial) 
communications course that an extraordinarily competent' 
contractor performed an analysis of decision-making and 
communicatioils in the contracting organization before 
designing the course. As a result the course appears to have^ 
met very real organizational needs. 

b. Much training resists effective evaluation because o^f a 
failure to state the purposes and intended outcomes of training 
in ways specific enough' to assess with confidence. 



In evaluating a course for scientists we found it difficult 
to assess the practical effectiveness of different aspects of 
the graining because specific behavioral objectives had not 
been defined for the many lectures and lab sessions which ^ 
constituted the course. Had objectives been clearly defined, . 
we could 'have drawn <aore conclusive inferences regarding the 
relatitre value each segment of training held for its partici- 
'pants, relying more upon performance than self-reporting. 

c. Evaluation is'usua'lly a one-tinid terminal event rather ^han 
a continuoi^s process that monitors «the effectiveness of 
training. 

In evaluating an executive development program^ we were *not 
able to begin until the program was completed. As a result we 
were not able to arrange for certain critical experimental 
controls that could h^ve allowed us far more confidence in 
generalizing from our findings. We were also limited by the 
data which had been collected prior to our evaluation. Although 
we collaborated in the design of comprehensive post-program 
instrumentation, it could not entirely compensate for the 
insufficiency- of the pre-program information. 

In addition to pre-training information and data which is 
collected during training, follow-up data is often an essential 
component of effective evaluation. This component requires 
strong management support sinde it typically involves a stddy 
of training impact of participant performance back on the job. 
Yet, without such information, the ifitimate effect of training 
on job performance and organizational productivity is difficult 
or impossible to assess. • 

d. Individual differences in the needs of various personnel 
for specific training have often been ignored, resulting in 
some individuals failing to get the training they need and 
others receiving training they do not need. 

In evaluating a technical/scientific course for* radionuclide 
material users, we discovered through pretesjting that several, 
molmbers of the class may have been able to pass the final . 
exam* without taking the course. As a result of similar past - 
experiences, trainers have sometimes attempted to screen 
trainees before training begins. However, these efforts have 
been largely countered by the view that prospectiMr^ trainees 
shouldn't be expected to give up work time in advance of 
training to respond to tests, assignments or interviews. 



S 



Management needs to ejicourage efforts to diagnose learner 
needs before training ao that pretesting has some priority iu 
the work setting. At the same time, trainers ^musc , have ready 
access tfo m^hpds for diagnosing trainee needs so £hat they 
possess sufficient: test and measurement methods to construct V 
eff icient mean^ for diagnosing needs. ; v • — 

In order to assure thkt training is undertaken dnly after an . 
analysis of performance proMems, that it-be carefully * " T*^ 
.planned to meet; clear performaace objectives, that it /include , 
continuous ervaluation, and that it fee given to those and only 
tl^<!)se who need it, a consolidation within agencies o£ training 
^ i^nd other staff activities directed toward improving employee 
-^effectiveness and productivity is warranted. 

Such a coordination 'or consolidation /of activities ijmplies as 
a minimum that some program analysts should ^have a b/slc 
knowledge of training principles-. It also 'implies that -ajialysts 
may qeed to work direq;:ly with trainers in, assessing j)erformance 
and recommending' Solutions to performance problems. The/ 
establishment of the Workforce Effectiveness and" Development 
Group within the 0PM serves as a model for such re'^tructuring. 
"As the WED group gains experience, we will •be ajile* tc provide' 
continuing guidance to consoiidatipn efforts.yCfirough research, V 
consultation and the publication of models ^d illust^^a««J^4 ^ 
standards* .... 

Recommendations: . 

0MB and 0PM, in all dealings with agencies, should reinforce* 

the practice of applying evaluation to all aspects of employee -^ " 

development, not just the actual classroom experience . . . 

/ OMB and 0PM should encourage coordination wlthin^agencies ' of ^ 
gaining and other stafT"activitie s directed toward improvi^ 
em'p loyee effec t iveness afnd productivity . ^ ~ ^ 

fhould j>rovlde g uidance and consti^tation concerning minimum 
essential training evaluation activities for operational settings . 

. . ' ■ ■ . ' ' ■ . * 

0PM s hould enc o urage agencies to use systeinatic program <levelopm^Ttf' 
processes for meeting new training needs . 

ssue 2: Many trainers lack either ^;ime, skill or incentives for 
conducting effective evaluation. ' » * 



Discussion: 

These are closely related problems- Evaluation is a speciali- 
zation within a demanding profession. While it is true that a 
competent professional trainer should be well versed in the 
basics of evaluation, it is unrealistic to expect most trainers . 
to be able to apply evaluative devices of the complexity or 
level of sophistication characterizing some of those employed 
in the demonstration project. 

In none of our evaluations did the host agency have the resources 
for performing the scope of assessment undertaken by the. 0PM 
evaluators. * ♦ . , ^ 

While most of these agencies have ^several training personnel, 
. in many organizations the small one or two person training shop 
barely able to keep ahead of the paper work is the 'norm rather 
than the exception. Individuals in these shops, even when 
capable of performing evaluation, would require a clear signai 
of changed -priorities before they could be expected to u^idettake 
even sporadic evaluations^ As it is, many government trainers 
have no relevant academic preparation for the profession. 
Short of requiring a -degree in a relevant subject for entry or 
advancement in the training profession, we can at least 
encourage better prepared people to enter ttte field, particularly 
those with preparation in* evaluation. 

Moreover, agencies may find it productive to analyze the tasks 
performed by trainers to see if a more efficient use can be made 
of their time and talents. Typically, professional-level 
trainers perform time-consuming admlnistative paper-work which 
may be accomplished by clerical personnel. The use of clerical 
personnel (or as appropriate, paraprof essionals) for these 
tasks would have the effect of allowing trainers to devote 
more attention to more appropriate activities such as assessment, 
analysis arid evaluation. 

Recommendations: 

0P M should Increase its active cooperation with colleges and 
universities in the development of programs which train 
specialists in evaluation for training and human performance . 
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0PM should encourage the inclu sion of employee development , 
evaluation and needs analysis as Important areas of management 
Concern and appropriate s ubjec ts of performa n ce staitdards apd . 
appraisal 'for trainers. 

Isstie 3t Muqh -contracted training is not carefully monitored for: 
effectiveness. * \ , 

Discussion: , 

In evaluating a contracted course 'for photocomposition machine 
operators, we found that the .contractor was allowed to state 
his own minimal« standards. Those standards did not specify 
criteria for measuring success on one of the three major 
learning ob^tives. As a result, degree of success 
was largely a matter 'of individual interpretation. 

Agency training officers need to know enough to judge whether a 
•contractor's evaluation methods are adequate^ In light of the 
limited expertise of trainers, guidelines in the form of sample 
V RFPs and sample training contracts may directly assist those 
responsible for purchasing training in assuring that the 
training will be evaluated on a clear set of criteria. A 
similar set of guides on testing for learning achievement 
may help the purchased to estimate the quality and adequacy 
of the contractor's proposed instrumentation. 

f ■ 

Recommendation: 

0PM should coordinate with the Office of Federal Procurement 
Policy in gathering information necessary for establishing 
policy and guidelines for agencies regarding the evaluation of 
contracted training . 

Issue 4: Training evaluation can be costly. It has to compete 

with other functions such as program administration and 
evaluation of other dimensions of performance; yet, 
little systematic cost data exists for the evaluation 
^ of training. 

Discussion: 

There has never been much cost-effectiveness data available in 
training. Such data presupposes amdng other conditions a very 
clear and detailed accounting of precisely what changes ensue 
in an organization's actual production, however it is measured. 
It al&o presupposes that tohe effects of training can be 



separated from 'a m^titude of other potential variables. 

The tralry^ng value models ^veloped by the Tralfilng Leadership 
Dlvlslon9*^^?M have been used increasingly over the last few 
years by a wide var^lety of agencies and other organizations. 
Their continued use and refinement should make them extremely 
useful In further cost-effectiveness studies. More evaluation 
demonstrations need to be conducted to provide a yet more 
representative body of examples, e.g., technical training 
courses for entry level personnel being px^epare^ for complex 
jobs such as Claims Representative or Benefits Authorlzer 
(such courses may currently run foV as fakny as 20 consecutive 
weeks); needs analysis studies; follow-up evaluations; and 
comparative studies of programs in different agencies with, 
similar objectives. 

Recommendations: • 

'■ ' • ^ 

0PM should provide guidance for deciding how to Invest 
training evaluation resources . 

0PM should encourage and cooperate with the Fed eral training 
conuau nity in performing training demonstration s in addition 
to^'those reported jiere . . \ ' , 

^Issue 5: There is a pervasive lack of Information regarding 
what people have learned in training and how well 
they have been able to apply that learning on the 

• job. ■ ■ ■ ■■,,_V. ' ■■ . 

Discussion: , 

A common reason given by agency trainers either for not main- 
taining a record of what trainees have learned or for falling 
to conduct follow-up studies of training Impact by reviewing 
performance with supervisors is that these. activities may 
violate the Privacy Act. >loreoyer, experimental programs in 
entry-level training are avoided out of concern over selection 
t^uldeilnes. In any case, there is a general lack of systematized 
iaEormatlon on the relationship of training with job performance. 

- ■■/ _ . . 

During our demonstration project, in evaluating the experimental 
course for photocomposition operators, we observed that agency 
personnel, despite their main Interest in the course's potential 
for training new operators, chose to give* it to a volunteer 
group of experienced operators. The agency personnel argued 
that placement decisions made from ^trainee performance in an 
experimental course coul^ be subject to grievance. Similarly, 



- in evaluating the. managerial course on coramunlcatj-ons, we'werta 
dissuaded from discussing specific trainee action plans with 
trainees' supervisors on the supposition that the^Prlvacy Act 
would be violated. ^ 

It would appear that agency managers and phe information officers 
. they consult need to give clear guidance rejgarding what can and 
c^nnotTBl^^e in the name of training and employee developmenf. 
F^urthermorie, these Individuals tHemselves need-^idance before • 
they caw set parameters riBgarding training information and the 
Privacy Act as veil as the Uniform Guidelines on Employee Selec- 
tion Procedures. # 

■ « /■ ■ * 

Within such parameters , the relationship' between training and " 

performance should be investigated more fully. AgenctW should 

strive for a system of heeds analysis and training, prescription 

which begj^ns with perf ormance appraisal. Appraisals^ should be 

studied and>used as the principal basis for training decisions. 

1 Following that, training should be designed to meet the specific 

job-related needis of the employee. : Finally , training ishould be 

assessed in relation to the subsequent job perfpridance of the 

employee as^ Reflected in the enduing performance appraisal. 

Recommendation: . ; 

' ■ " ■ » — 

QPM stiould develop guidance for agencies to assist them to 

obt ain group perfomiance Information without violating laws : 

or other pr otections to employees . 

~ : — , 

0PM should encourage the use of performance appraisals as 
vehicles for measuring the need for training and the effects 
of training . on performance . ; 

Issue 6: Recommendations to evaluate training or human performance 
may be ignored by decision-makers in their efforts to 
give full attention to program implementation. Further- 
more, recommendations coming from evaluations may be 
similarly Ignored. Thus however competent the evaluation 
it has no value if it' is not used. 

Discussion: 1 

Many managers are unaware of the very substantial advances that 
have been made in the immediate past in the technology of 
evaluation, including techtiiques^to project the costs and 
benefits of training. Consequently, they are either ignoring 
the evaluation process or they are accepting evaluation and 
projections much less rigorous than what the present state of 
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the art is capable of producing. 



une literature on evaluation research documents the commonality 
If decision-makers disregarding the fijidings. conclusions and 
recommendations of evaluations. Often, this disregard stems 
from the decislon-Tnaker's perception that an evaluator mlsunder 
stood the problem, used methodology inappropriate to ^he task, - 
or failed to properly consider organizational constraints and 
imperatives in miking recommendations. In other Instances, 
disregard for evaluation information appears to stem largely 
from a commitment to the status quo.''" . 

Recommendat ign : 

0MB sh ould require supportive analysis and justif Icatlon for- ^ 
lorg. «ra1P Investme nts in t ralnlne (including trainee salaries ) 
Vn _H^ «an.e wav that jusyif TTation is required for any, other 
discrete expenditure . 
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Ah Evaluation of the Executive Development vPrograra 
of the Science and Education Administration, United 
' * ^ States Department of Agriculture 

Executive Siimmary 

The' Agriculture Research unit of U.S.D.Ai's Sclenpe and Education 
Administration '(SEA) Is charged with assuring the availability 
of an adequate supply of moderately priced agriculture commodities^ 
to the marketplace; Accomplishment of this mlfislon is based, in 
part, on successful scientific research and effective management 
of , the research function. \ , - ^ ^ 

In prder to prepare potential executives to manage this function, 
an executj.ve development program was designed for SEA and imple- 
mented in June of 1977. That program became the object of the - 
evaluation demonstration begun in September of 197B. ' 

Three major objectives were defined for the evaluation" by 0PM 
and SEA staff: - . . ; 

EVALUATION OBJECTIVE 1 

Determipe the extent to which potential executives have 
develolecfi/professlonally during the Executive Development 
Program; ^ . . / 

EVALUATION OBJECTIVE 2 ' 
Determine the role of the mentor^nderstudy -relationship 
in the development of the potential executives f 

EVALUATION OBJECTIVE 3 

Determine ways in which the Executive Development Program 
can be Improved. . 

' ^ r ^* * ■ ■ , 

Several sources/ of data were employed in the evaluation. Data for 
Objective 1 were gathered from XD Program participants called . 
Potential Executives (PEs) and senior managers who served as 
their mentors. Both groups were asked to write narrative reports 
on their experiences in the XD Program. In addition^ PEs were 
asked to send in copies of Executive Work Assignmeiita and mentors 
were asked to send in completed copies of "The Appraisal 0f 
Potential for Management" form. All of the above were collected 
in :^yemb4r, 1978. A s^tond phase of data gathering related to 
the*i^£i|j! pT growth Involved the group of PEs, a 

compa^ife|ii" gro runner s-up to the program, and a selected 

group ofvl^^ executives. These Individuals completed the 

"Appraisal Executive Attributes"* questionnaire. Their response 



were collected at the beginning of January, 1979. Prograo^ ^ 
participants appeared to experiei^ce significantly more professional 
growth during the 18 month program than did, their comparison group 
counterparts. ^ 

Data for Objective 2 were supplied in November, 1978, by PEs and 
mentors using a questionnaire on the mentor process. Regarding/ 
the second objective, while there were areas for improvement, - 
participants and their mentorA agreed that their relationships 
were uae^ul and^enhauced the effectiveness of the program. 

Data for Objective 3 wete gathered from PEs and mentors in. 
November, 1978, in the forji of narrative critiques of the XD 
Program.-' With respect to Objective 3, 'major suggestions for 
improving the program involved the dreation of more "acting f 
positions outside the participants' i&mediate areas of work I 
and a lengthening of the program. ° . 

As a result of the findings, the evaluation team recommended 
that the Executive Development i^^rogram be c^jjlfljaued. In 
addition, it recommended that the mentor-understudy relation- - 
ship be continued with modifications which would ensure appro-r 
priate matching of potential executives and mentors and which 
would encourage cpntinued improvement of this aspect of- the 
program. The evaluation team also recommended that an on- 
going management system, for the program be de'signed and 
implemented as a means of ensuring that the substance of the 
program remai-ns appropriate to the needs of participants and 
the agency. * ' 
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An Evaluation of a Scientific/Technical Course 
on Radiological Health and Safety Presented by 
^ ^ the National Institutes of Health 



Execiitive Summar 



Research efforts at\ the National Institutes of Health (NIH) include 
the use of radioactive material, both in laboratory work and* in 
the treatment patients. As part of the Nuclear Regulatory 
Commission requirements, organizations such as NIH which are 
licensed to use radionuclides must take a number^f safety 
precautions. For instance, in order for NIH personnel to be 
authorized to order radioactive material, th^y must demonstrate 
that t\xey h^ve appropriate experience with and knowledge of its 
use. >One means of meeting the knowledge requir^enf is to 
successfully complete a nine-day course offered by NIH, entitled 
"Radiological Health for Radionuclide Users." 

The course is conducted by NIH' s' Radiation Safety Branch (RSB), 
which has a variety of radiation safety responsibilities. Sixty, 
people attend each session of the course,^which Is offered twice 
a^year. The course faculty includes NIH personnel and guest 
lecturers. The goals of the course are to: 1) enable NIH staff 
to become authorizefi users of radioactive materials; 2), give 
participants a working knowledge of radiological health so that 
they can evaluate hazards on the job; 3) instruct participants 
in specific tools and techniques for carrying out research efforts 
which involve radioactive material; and 4) provide information 
about the staff,** services, and sources of assistance in the 
Radiation Safety Branch. 

Course evaluations in the past had' consisted of obtaining 
participants' opinions at the end of the course. For this 
e\^aluation effort, a variety of data was sought which could 
be used to further judge course effectiveness and make changes 
In the course if appropriate. The evaluation t>bjectives were to: 

1. Determine the adequacy and congruence of the course 
process (sequence, instructional strategies, etc.)« 

2. Determine participants • perceived value of the course. 

3. Determine participants' degree of learning. 

4. Determine perceived change* in behavior as a result 
of the course. ^ 

5. Determine the costs of the course. / 

Measures were taken during one course session, as well as six montl 
aftfer the completion of an earlier comparable course session. The 
evaluation of the course Indicated that all four cpurs^ goals were 
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met. Moreyover, 80 percent of the participants said that they would 
recommend the course to others even if it were not required to 
become an afltithorized user. 

The evaluators noted, however, that portions of the course may 

not be necessary for certain participants. One reason for, this 
state of affairs is that a single course addresses two audiences 
with different sets of needs— the Ph.D. laboratory researchers, 
and the.M.D. clinician. The Urst group uses radioactive 
material as part of its experiments, and exposure Is certainly 
unplanned and undesired. the second group, the M.D.s, while 
oeeding to protect themselves from exposure, deliberately intro- , 
duce a defined amount of radioactive material into humans. While 
both groups may need a common core of inforination in the Radio- 
logical Health course j they also have distinct knowledge 
requirements. . 

Suggges'tions were presented to NIH for making the course more ' 
efficient. The possible changes to the course include: 

1) dropping, shortening, and/or combining certain lectures; 

2) providing core informatioif, then having two independent- 
tracks, one for laborato^ry researchers and one for clinicians; 

3) providing two different courses for the two primary audiences; 

4) designing the course in modules to meet individual needs. 

Finally, it was suggested that instead of, or in addition to, pro- 
viding instruction during the nine-day course on specific research 
techniques involving the use of radionuclides, separate short 
courses on Chose techniques might prove worthwhile. 
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An Evaluation of a Technical Course on Electronic 
Photoconjiposition Keyboard Techniques Presented at^ 
the Government Printing Office .— 

\ ■ 

Executive Summary ^ 

The Electronic Photocomposition Division (EPD) of the Government 
PrinJ^ng Office will be assimilating large humbersr of entry-leve|^ 



. keyl^tird operators ih the near future. At the s^amie tim^, as 
olde^^typesetting processes are being phased out, the workload-^ 
demahds on EPD are growing. A^'concern for this situation led 
EPD to look fot new ways tq train both entry-leyel and journeyman- ♦ 
level' operators in keying techniques. The Malt Audio-Visual 
Instructional System (MAVIS) was viewed at a trade show and * 
subsequently contracted\ f or a pilot program to train 12 journeyman- 
level operators. Three 'major objectives were established* for 

the training by thecontractor: 

■ ^ ■■ 

1. Increase typing speed from 15 percent, to 40 
percent on an average across the group. ^ 

2. Decrease errors 50 percent on an average across 
the group. * ' 

3. Decrease^the fatigueithat comes from typing all day. * ' 

In light of these training objectives, the 0PM evaluation team 
set three evaluation objectives: 

1. To measure whether the training accomplished its " / 
objectives.. • 

2. To measure the economic impact of the training. . 

3. To recommend improvement for the training process. 

*The evaluation design used was a modified pre-control and post- 
, control group design. Pre-training and post-training production 
and accuracy measures were taken,, trainee and control group 
behavior was observed, and trainee. perceptions were solicited. ^ 
This design yieldejd information shb^ing that only the training v ' 

objective on fatigue was successfully met but that the training 
did seem to produce tangible benefilrts to the organization. 
Several recommendations were made concerning the improvement of " 
the training and the evaluation. 

This effort demonstrates the feasibility and practicality of j 
evaluating the effectiveness of technical training as well as 
some of the difficulties which are* inherent in trying" to use 
an experimental design in an operational setting. ^ 

18 / . • * 
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^ ■ An Evaluation of an Interagency Course on \^ 

Training Evaluation Presented by the Office 
■ ^ ' of Personnel Management 

r . ( 

Executive Sunngary ^ ^ , . 

Through a trllnlng needs survey, the Personnel^Management Training 
Center of the U. S. Office of Personnel Management Identified a 
training need shared by many Employee Development Specialists In 
the area of training, evaluation. To mAetfhls need, a course based 
on the publication AProcess for the Evaluation of Training was 
developed and conducted. That course, "Evaluation of Training 
Courses," was selected to be the subject of this demonstration 
project. 

'The Instructor's primary course objectlv^ was to transfer evaluation 
technology as exemplified In the Process to agency trainers.* The 
evaluation Instruments used were designed to elicit Information' which 
would help make the course more jresponsivl to agency need* consistent 
with this objective. The following evaluation objectives were formu- 
lated during preparation of an evaluation plan by the InsWvlctor: 

1. Determine whether piartlclpants are able to meet course 
objectives, and. If not, why not, V 

2. Ascertain participants' . reaction to content appropriateness,, 
presentation effectiveness, written materials effectiveness, 
personal accomplishment of objectives, and Instructor effec- 
tiveness, 

- • . 

3. Determine whether skills and/or knowledges acqulted during 
J:ralnlng are used back on the job, and. If not, why not. 

A review of the major findings In relation to these evaluation 
objectives disclosed the fact that most participants were 
able to meet course objectives. Indicating tha^ considerable > 
learning took place. Participants rated the course favorably. 
However, -participants appeared to experience difficulty in 
applying knowledge of evaluation design and data analysis 
as taught in the course. ''They cited primary obstacles as their 
ownJ.ack of time for performing evaluation and lack of management 
support for evaluation activities. This latter constraint 
is seen as an indication that managers do not yet perceive 
training evaluation as a useful tool in performance Improvement 
and human development. 
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An Evaluation of a Clerical Course oh Travel 
Regulations Presentedr by tHe National Aeronaiitlcs 
and Space Adralhliittiatlon 

Executive Summary 

.. ^ 

Pecsonnel at the Financial Management Office (BFH),^e NASA 
Headquarters office which Is re8pon%lble for processing travel 
forms and certifying travel vouchers for jfeyment, detected a per- 
formance problem. They noted that many travel forms were sub- 
mitted Incotrectly ^ BFH and that some NASA secretaries called 
BFH regularly for Information about b^^c travel-' regulations, 
slowing down tfie BFH work process. 

BFH staff therefore requested NASA's Headquarters training office 
to develop and offer a course Tor NASA secretaries on the regula- 
tions and policies governing reimbursement for domestic travel. 
Such a course was developed aiid given twice, ^nce In May, 1978, 
an4 again In November, 1978. The second session of the course 
was evaluated by 0PM personnel working In conjunction with the 
course Instructor. 

^e objectives of the evaluation were designed to address both 
the Internal Integrity of the^trainlng and Its organizational 
Impact. These evaluation objectives were: ^ 

1% Assess trainee classroom performance on the course 
objectives. 

2. Assess the d(ongruence of the course process with the 
course objectives. 

3. Estimate the value of ^ gaining to the organization. 



Identify factors related to performance which are 
not affected by the course. 



Data was gathered through an examination of work samples, pretests 
and posttests, observations ^ Interviews, and questionnaires. Some 
of the data used In determining the value of training were estimates; 
therefore the cost-effectiveness figures are approximate rather than 
precise. . 

The evaluation team found that trainees by and large met the 
course objectives and that the overall course process was congru- 
ent with thiB objectives. However, although the training was 
effective In transmitting learning. It did not appeiar to be 
cost-effective to the organization. The dollar value of the time 
losjt: through the performance deficiency seems to be less than the 
cost of graining to eliminate the deficiency. Furthermore, examination 
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of other performance-related variables revealed that other solutions 
to the performance problem might be more cost-effective than formal 
piafbsroom training. Recommendations*' were that NASA consider 1) non- 
classroom methods \of acquiring. sJcHls and knowledges, and 2) alternative 
ways of organizing the work flow associated with "travel' forms. 
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POi Evaluation of An Organizational Development 
ani Training &f fort in Management Communications 
Xonduct at. Fort Belvoir . 



' * . • Executive Summaty ^ _ ^ 

When two organizational components' depend upon one another for infor- 
.mat ion. they require to accomplish their missions, they share a need 
* for communication which is timely,' clear, apd adequate. Such inter- 
dependence is charafcteristic of two directorates at the Fort Belvoir 
Engineering Center, Department of the Army* 

• ■ ■ . ■ , ■ - • ' / / ' 

The Fort Belvoir Post Commander and his staff perceived communication 
between th^se two directprates as disruptive, of mission accomplishment 
Vather than facilitative.V Command concern led to a contract for. a . 
•^managerial communications" course for directorate managers. That 
course became the subject of the evaluation discussed here. 

The course was explicitly designie^d to help managers communicate 
management needs and management information to one another, trf sub- 
ordinates and to superiors, both within their own directorates and 
across directorates. In light of the purpose of training, three 
major ofejectl^es were formulated for the evaluation: 

1. To assess on-the-job changes in participant; 
behavior*' which resulted from the training. 



2. To assess the effect of the training on the 
two-directorate^. # 

' ' 3. To identify problems encountered when 
' ^ participants attempted* behavior changes- 
. ;.ahd problems or concerns of iridi vidua J.s who 
* Wete directly or* i^^ 
parti'cipantsi behayio^r^ . 

An evaluation technique 'which; fpcU and behavior 

; change was used ttf collect mcist j>f thetdata (th 
Plan Approach). It yielded a clear picture pf improved communications 

•practices among course participants as. well as^esolutions of work 
I^robiems within andy acroas 4iriector^f^.; iFurthermore, no new problems 
were perceived to have resulted from actions the participants 
initiated on-the-job to Improve communications. - , 

M'?: • . ; ^ ' , ' > ' 

Thus;' the cour conf ir^ed a? an e^^ means for initiating 

' the iiaprovemenfc of ihterdirectotati/Vpmmunlc^ Furthermore, the 

^valuators rebomraended; that, rt^^^^^^ by organizational 

dev^lopm^ht efforts at! other ieveli within the 

Command', to Include other course, se&s^^^^ managers ^who could not 
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attend the first session. In addition," the evaluators recommended 
. that "action planning" (participants setting performance objectives 
for themselves) be employed a^ a major means of Inducing Improvement 
in comtaunlcat Ions behavior. - 
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